Imaging device, solid-state image sensor, camera module, drive control unit, and imaging method

ABSTRACT

The present disclosure relates to an imaging device, a solid-state image sensor, a camera module, a drive control unit, and an imaging method by which an effect of motion on an image can be reliably corrected. The drive control unit controls drive of at least one of an optical system or an imaging unit by finding, on the basis of motion of the imaging unit that is physically detected, an amount of movement at the time at least one of the optical system or the imaging unit is moved relative to another to optically correct blurring appearing in the image captured by the imaging unit. A signal processing unit performs signal processing for correcting an effect of the motion of the imaging unit on the image according to a position conversion function based on position information and motion information synchronized with each coordinate on the image, on the basis of the position information obtained by detecting a position of the optical system or the imaging unit driven according to the control by the drive control unit and the motion information indicating the motion of the imaging unit physically detected. The present technology can be applied to a stacked CMOS image sensor, for example.

TECHNICAL FIELD

The present disclosure relates to an imaging device, a solid-state imagesensor, a camera module, a drive control unit, and an imaging method,and more particularly to an imaging device, a solid-state image sensor,a camera module, a drive control unit, and an imaging method by which aneffect of motion on an image can be reliably corrected.

BACKGROUND ART

Optical image stabilizer (OIS) or electronic image stabilization (EIS)has been used as a technique for correcting camera shake in an imagingdevice. The optical image stabilizer can correct blurring by moving oneof a lens and an imaging element relative and parallel to anotherdepending on an amount of blurring and shifting the position of an imageon the imaging element. The electronic image stabilization can correctblurring by cutting an image captured by an imaging element as an outputimage and shifting the cut position depending on an amount of blurring.

Now, actual camera shake is mainly caused by rotational motion and isless influenced by parallel movement where, in particular, the influenceof parallel movement decreases as the distance to a subject increases.The technique of optical image stabilizer corrects the rotational motionby parallel movement of the lens or imaging element, so that the edgemay be deformed in some cases. Similarly, the electronic imagestabilization performs correction that causes parallel movement of thecut position, thereby having a problem that the edge is deformed.

Furthermore, no measure has been taken against deformation (a focalplane phenomenon) caused by a difference in the amount of movementwithin one screen due to a gap in exposure time for each pixel line thatoccurs in an imaging element using a rolling shutter such as acomplementary metal oxide semiconductor (CMOS) image sensor.

Thus, as disclosed in Patent Document 1, there has been proposed animaging device that can perform image stabilization while accommodatinga difference in the amount of movement due to the position within animage plane and a difference in the amount of movement due to a gap inexposure time within one screen. Adoption of such image stabilizationcan correct camera shake very accurately from the center to the edge,and can also correct deformation caused by the focal plane phenomenon.

CITATION LIST Patent Document

-   Patent Document 1: International Publication No. 2014/156731    pamphlet

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

Now, in a case where imaging is performed with a short exposure timesuch as in the daytime outdoors, the effect of camera shake is almostcompletely corrected by performing the image stabilization of PatentDocument 1 above so that the occurrence of blurring and deformation inan image can be reduced. However, in a case where imaging is performedwith a long exposure time such as in a dark place or at night, theoccurrence of blurring and deformation in an image can be corrected byperforming the image stabilization of Patent Document 1 above, but it isdifficult to reduce the occurrence of blurring of a point image duringexposure (hereinafter referred to as blurring within the exposure time).

The present disclosure has been made in view of the above circumstances,and an object of the present disclosure is to reduce the occurrence ofblurring within the exposure time and be able to reliably correct theeffect of motion on an image.

Solutions to Problems

An imaging device according to one aspect of the present disclosureincludes: an imaging unit that captures an image of a subject via anoptical system collecting light from the subject; a drive control unitthat controls drive of at least one of the optical system or the imagingunit by finding, on the basis of motion of the imaging unit that isphysically detected, an amount of movement at the time at least one ofthe optical system or the imaging unit is moved relative to another tooptically correct blurring appearing in the image captured by theimaging unit; and a signal processing unit that performs signalprocessing for correcting an effect of the motion of the imaging unit onthe image according to a position conversion function based on positioninformation and motion information synchronized with each coordinate onthe image, on the basis of the position information obtained bydetecting a position of the optical system or the imaging unit drivenaccording to the control by the drive control unit and the motioninformation indicating the motion of the imaging unit physicallydetected.

A solid-state image sensor according to another aspect of the presentdisclosure includes: an imaging unit that captures an image of a subjectvia an optical system collecting light from the subject; and a logicunit that performs processing of adding, to the image captured by theimaging unit, position information obtained by detecting a position ofthe optical system or the imaging unit driven according to control by adrive control unit and motion information indicating motion of theimaging unit physically detected, and outputs the image to which theposition information and the motion information is added to a signalprocessing unit that performs signal processing for correcting an effectof the motion of the imaging unit on the image according to a positionconversion function based on the position information and the motioninformation synchronized with each coordinate on the image on the basisof the position information and the motion information, the drivecontrol unit controlling drive of at least one of the optical system orthe imaging unit by finding, on the basis of the motion of the imagingunit that is physically detected, an amount of movement at the time atleast one of the optical system or the imaging unit is moved relative toanother to optically correct blurring appearing in the image captured bythe imaging unit.

A camera module according to another aspect of the present disclosureincludes: an optical system that collects light from a subject; animaging unit that captures an image of the subject via the opticalsystem; a drive control unit that controls drive of at least one of theoptical system or the imaging unit by finding, on the basis of motion ofthe imaging unit that is physically detected, an amount of movement atthe time at least one of the optical system or the imaging unit is movedrelative to another to optically correct blurring appearing in the imagecaptured by the imaging unit; and a logic unit that supplies positioninformation, motion information, and timing information indicating atiming for synchronizing the position information and the motioninformation with a coordinate on the image to a signal processing unittogether with the image captured by the imaging unit, the signalprocessing unit performing signal processing for correcting an effect ofthe motion of the imaging unit on the image according to a positionconversion function based on the position information and the motioninformation synchronized with each coordinate on the image on the basisof the position information obtained by detecting a position of theoptical system or the imaging unit driven according to the control bythe drive control unit and the motion information indicating the motionof the imaging unit physically detected.

A drive control unit according to another aspect of the presentdisclosure controls drive of at least one of an optical system or animaging unit by finding, on the basis of physically detected motion ofthe imaging unit capturing an image of a subject via the optical systemcollecting light from the subject, an amount of movement at the time atleast one of the optical system or the imaging unit is moved relative toanother to optically correct blurring appearing in the image captured bythe imaging unit, and supplies position information, which is obtainedby detecting a position of the optical system or the imaging unit drivenaccording to the control, and motion information, which indicates thephysically detected motion of the imaging unit, to a logic unit thatperforms processing of adding the position information and the motioninformation to the image captured by the imaging unit and outputs theimage to which the position information and the motion information isadded to a signal processing unit that performs signal processing forcorrecting an effect of the motion of the imaging unit on the imageaccording to a position conversion function based on the positioninformation and the motion information synchronized with each coordinateon the image on the basis of the position information and the motioninformation.

An imaging method according to another aspect of the present disclosureincludes the steps of: controlling drive of at least one of an opticalsystem or an imaging unit by finding, on the basis of physicallydetected motion of the imaging unit capturing an image of a subject viathe optical system collecting light from the subject, an amount ofmovement at the time at least one of the optical system or the imagingunit is moved relative to another to optically correct blurringappearing in the image captured by the imaging unit; and performingsignal processing for correcting an effect of the motion of the imagingunit on the image according to a position conversion function based onposition information and motion information synchronized with eachcoordinate on the image on the basis of the position informationobtained by detecting a position of the optical system or the imagingunit driven according to the control and the motion informationindicating the physically detected motion of the imaging unit.

According to one aspect of the present disclosure, the drive of at leastone of the optical system or the imaging unit is controlled by finding,on the basis of the physically detected motion of the imaging unitcapturing the image of the subject via the optical system collectinglight from the subject, the amount of movement at the time at least oneof the optical system or the imaging unit is moved relative to anotherto optically correct blurring appearing in the image captured by theimaging unit. Then, the signal processing for correcting an effect ofthe motion of the imaging unit on the image is performed according tothe position conversion function based on the position information andthe motion information synchronized with each coordinate on the image,on the basis of the position information obtained by detecting theposition of the optical system or the imaging unit driven according tothe control by the drive control unit and the motion informationindicating the motion of the imaging unit physically detected.

Effects of the Invention

According to one aspect of the present disclosure, the effect of motionon an image can be reliably corrected.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an example of distortion occurring inan image subjected to the effect of lens distortion.

FIG. 2 is a graph illustrating vibration conditions applied to animaging device.

FIG. 3 is a diagram illustrating an example of an image that is outputwithout blurring being corrected.

FIG. 4 is a diagram illustrating an example of an image on whichcorrection processing is performed according to normal electronic imagestabilization.

FIG. 5 is a diagram illustrating an example of an image on whichcorrection processing is performed according to optical imagestabilizer.

FIG. 6 is a diagram illustrating an example of an image on whichcorrection processing is performed according to image stabilizationproposed in Patent Document 1.

FIG. 7 is a diagram illustrating an example of an image on whichcorrection processing is performed according to image stabilizationproposed in Patent Document 2 and in which a correction is not performedagainst lens distortion itself.

FIG. 8 is a diagram illustrating an example of an image on whichcorrection processing is performed according to image stabilizationproposed in Patent Document 2 and in which a correction is performedagainst lens distortion itself.

FIG. 9 is a diagram illustrating an example of an image that is outputwithout blurring being corrected in a case where imaging is performedwith a long exposure time.

FIG. 10 is a diagram illustrating an example of an image on whichcorrection processing is performed according to image stabilizationproposed in Patent Document 2 and in which a correction is made againstlens distortion in a case where imaging is performed with a longexposure time.

FIG. 11 is a diagram illustrating an example of an image on whichcorrection processing is performed according to optical image stabilizerin a case where imaging is performed with a long exposure time.

FIG. 12 is a diagram illustrating an example of an image on whichcorrection processing is performed according to image stabilization byan imaging device to which the present technology is applied.

FIG. 13 is a block diagram illustrating an example of the configurationof a first embodiment of an imaging device to which the presenttechnology is applied.

FIG. 14 is a graph for explaining correction processing performedaccording to electronic image stabilization by a signal processingcircuit.

FIG. 15 is a flowchart for explaining image stabilization processingexecuted in an imaging method employed by the imaging device.

FIG. 16 is a graph for explaining a result of image correction.

FIG. 17 is a block diagram illustrating an example of the configurationof a second embodiment of an imaging device to which the presenttechnology is applied.

FIG. 18 is a diagram for explaining OIS control information.

FIG. 19 is a diagram for explaining processing of returning a controlposition of optical image stabilizer to a center between frames.

FIG. 20 is a diagram for explaining processing of returning a controlposition of optical image stabilizer toward a center between frames.

FIG. 21 is a diagram for explaining blurring that occurs in a stillimage.

FIG. 22 is a diagram illustrating an example of an image on whichcorrection processing is performed according to image stabilizationproposed in Patent Document 2.

FIG. 23 is a diagram illustrating an example of an image on whichcorrection processing is performed according to image stabilization bythe imaging device in FIG. 13.

FIG. 24 is a diagram for explaining blurring that occurs in a stillimage by larger vibration.

FIG. 25 is a diagram illustrating an example of an image on whichcorrection processing is performed according to image stabilizationproposed in Patent Document 2.

FIG. 26 is a diagram illustrating an example of an image on whichcorrection processing is performed according to image stabilization bythe imaging device in FIG. 13.

FIG. 27 is a diagram illustrating an example of an image on whichcorrection processing is performed according to image stabilization bythe imaging device in FIG. 17.

FIG. 28 is a diagram illustrating definitions of a pitch direction, ayaw direction, and a roll direction.

FIG. 29 is a diagram illustrating an example in which an image sensor isused.

FIG. 30 is a diagram illustrating an example of a schematicconfiguration of an endoscopic surgical system.

FIG. 31 is a block diagram illustrating an example of a functionalconfiguration of a camera head and a CCU.

FIG. 32 is a block diagram illustrating an example of a schematicconfiguration of a vehicle control system.

FIG. 33 is an explanatory diagram illustrating an example of theinstallation position of each of an extra-vehicle information detectingunit and an imaging unit.

MODES FOR CARRYING OUT THE INVENTION

First, before describing an imaging device to which the presenttechnology is applied, vibration and image stabilization processing ofan imaging device will be described with reference to FIGS. 1 to 12.

<Regarding Vibration and Image Stabilization Processing of ImagingDevice>

FIG. 1 illustrates an example of distortion occurring in an image due toan effect of lens distortion when a subject is imaged by an imagingdevice.

For example, when a lattice pattern as illustrated in A of FIG. 1 isimaged as the subject, pincushion distortion with the edge portionshrinking inward as illustrated in B of FIG. 1 or barrel distortion withthe edge portion stretching outward as illustrated in C of FIG. 1occurs.

The following description describes blurring that occurs in an imagewhen the imaging device with the pincushion lens distortion asillustrated in B of FIG. 1 performs imaging while applying vibrationunder vibration conditions (the shake angle in a yaw direction: 1.5degrees, and the shake angle in a pitch direction: 1.2 degrees) asillustrated in FIG. 2. The left side of each of FIGS. 3 to 12illustrates four images captured in the vicinity of the minimum shakeangle (for example, two points where the shake angle is 0 degree) and inthe vicinity of the maximum shake angle (for example, two points wherethe shake angles in the yaw direction are 1.5 degrees and −1.5 degrees)in one cycle of vibration illustrated in FIG. 2. Moreover, the rightside of each of FIGS. 3 to 12 illustrates an image obtained bysuperimposing these four images.

FIG. 3 illustrates an example of an image that is output without beingsubjected to correction processing of correcting blurring with respectto vibration.

As illustrated in FIG. 3, various deformations depending on thepositions within an image plane occur due to the effects ofmisalignment, edge deformation, and rolling shutter caused by vibrationof the imaging device.

FIG. 4 illustrates an example of an image on which correction processingis performed according to normal electronic image stabilization. Here,the normal electronic image stabilization is to correct blurring bycutting an image captured by an imaging element as an output image andshifting the cut position depending on the amount of blurring, and isdifferent from correction processing of Patent Document 1 and PatentDocument 2 as described later.

As illustrated on the left side of FIG. 4, the correction processing isperformed according to normal electronic image stabilization to outputimages with the cut positions shifted depending on the amount ofblurring. The image is thus corrected such that, when these images aresuperimposed, the positions of the images coincide at the center of ascreen among frames as illustrated on the right side of FIG. 4. However,the correction processing according to the normal electronic imagestabilization cannot correct the effect of rolling shutter and the edgedeformation caused by camera shake.

FIG. 5 illustrates an example of an image on which correction processingis performed according to optical image stabilizer.

As illustrated on the left side of FIG. 5, the correction processing isperformed according to optical image stabilizer to output images thatare captured by moving one of a lens and an imaging element relative andparallel to another depending on the amount of blurring. Thus, whenthese images are superimposed, as illustrated on the right side of FIG.5, the positions of the images among frames at the center of a screencan be corrected and at the same time the effect of rolling shutter canbe corrected. In this case, however, the edge deformation due to camerashake cannot be corrected.

Note that with optical image stabilizer (employing the barrel shiftmethod or sensor shift method), the effect of edge deformation and lensdistortion remains, but the occurrence of blurring within the exposuretime can be reduced by performing the correction processing to followvibration even during exposure.

FIG. 6 illustrates an example of an image on which correction processingis performed according to image stabilization proposed in PatentDocument 1 described above. In addition to shifting the cut position asin the case of the normal electronic image stabilization, the imagestabilization disclosed in Patent Document 1 performs deformation foreach pixel position according to a difference in the amount of movementdue to the position within the image plane and a difference in theamount of movement due to a gap in exposure time within one screen.

As illustrated in FIG. 6, the correction processing according to imagestabilization proposed in Patent Document 1 is performed to reliablycorrect blurring from the center to the edge of the image. Note that thecorrection processing does not consider the effect of lens distortion,whereby an actual imaging result has an error due to the effect of lensdistortion and some misalignment occurring in the edge portion. Notethat this misalignment varies depending on the shape of the lensdistortion.

Now, as previously filed as PCT/JP2016/070261 (hereinafter referred toas Patent Document 2), there is proposed correction processing that canperform image stabilization in consideration of the effect of lensdistortion.

An image on which the correction processing is performed according tothe image stabilization proposed in Patent Document 2 will be describedwith reference to FIGS. 7 and 8.

FIG. 7 illustrates an example of an image on which the correctionprocessing is performed according to the image stabilization of PatentDocument 2 and in which a correction is performed against deformationdue to camera shake caused by the effect of lens distortion but is notperformed against the lens distortion itself.

As illustrated in FIG. 7, the correction processing according to theimage stabilization proposed in Patent Document 2 is performed toreliably correct blurring from the center to the edge of the image.

FIG. 8 illustrates an image on which the correction processing isperformed according to the image stabilization of Patent Document 2 andin which a correction is performed against deformation due to camerashake caused by the effect of lens distortion and is also performedagainst the lens distortion itself.

As illustrated in FIG. 8, the correction processing according to theimage stabilization proposed in Patent Document 2 is performed toreliably correct blurring from the center to the edge of the image withthe lens distortion being corrected.

Now, in a case where imaging is performed with a short exposure timesuch as in the daytime outdoors, the correction processing according tothe image stabilization proposed in Patent Document 2 can almostperfectly perform image stabilization. On the other hand, in a casewhere imaging is performed with a long exposure time such as in a darkplace or at night, blurring within the exposure time occurs.

FIG. 9 illustrates an image that is output without blurring beingcorrected as in FIG. 3 in a case where imaging is performed with a longexposure time.

As illustrated in FIG. 9, imaging performed with a long exposure timecauses blurring within the exposure time in addition to variousdeformations depending on the positions within an image plane due to theeffects of misalignment, edge deformation, and rolling shutter.

FIG. 10 illustrates an image on which the correction processing isperformed according to the image stabilization of Patent Document 2 andin which a correction is performed against deformation due to camerashake caused by the effect of lens distortion and is also performedagainst the lens distortion in a case where imaging is performed with along exposure time.

Even with the correction processing according to the image stabilizationproposed in Patent Document 2, camera shake is not reliably corrected inthe image due to the occurrence of blurring within the exposure time,though the positions of images coincide after the correction. In otherwords, while the blurring is successfully corrected from the center tothe edge of the image with the lens distortion being corrected, theimage has blurring within the exposure time.

Moreover, FIG. 11 illustrates an image on which correction processing isperformed according to the optical image stabilizer (employing thebarrel shift method or sensor shift method) in a case where imaging isperformed with a long exposure time.

As illustrated in FIG. 11, the correction processing according to theoptical image stabilizer cannot correct the edge deformation due tocamera shake but can reduce the occurrence of blurring within theexposure time by moving a lens and an imaging element relative to eachother even during the exposure time.

Accordingly, the applicant of the present application proposescorrection processing that more reliably corrects image blurring byreducing the effect of camera shake on an image even in a case of a longexposure time, as with an imaging device 11 of FIG. 13 and an imagingdevice 11A of FIG. 17 as described later. This correction processing canreliably correct blurring from the center to the edge of an image withlens distortion being corrected, and can also reduce the occurrence ofblurring within the exposure time.

In other words, as illustrated in FIG. 12, the correction processingapplying the present technology can reliably correct blurring from thecenter to the edge of an image with lens distortion being corrected, andcan also reduce the occurrence of blurring within the exposure time.Note that exposure blurring due to edge deformation during the exposuretime cannot be reduced by the optical image stabilizer, so that theblurring within the exposure time is expected to remain a little at theedge of the image depending on the vibration conditions and exposuretime, but can be reduced to an extent that it is almost unnoticeable innormal imaging.

Furthermore, FIG. 12 illustrates an image in which a correction isperformed against lens distortion itself as in FIG. 8. Note thatalthough not shown in the figure, even in a case where a correction isnot performed against the lens distortion itself as in FIG. 7, theblurring can be reliably corrected from the center to the edge of theimage and at the same time the occurrence of blurring within theexposure time can be reduced.

<Example of Configuration of Imaging Device to which Present Technologyis Applied>

A specific embodiment to which the present technology is applied willnow be described in detail with reference to the drawings.

FIG. 13 is a block diagram illustrating an example of the configurationof a first embodiment of an imaging device to which the presenttechnology is applied.

As illustrated in FIG. 13, an imaging device 11 includes a lens unit 12,an image sensor 13, a motion sensor 14, an OIS driver 15, an OISactuator 16, a signal processing circuit 17, a display 18, and arecording medium 19.

The lens unit 12 includes one or a plurality of lenses to collect lightfrom a subject and form an image of the subject on a sensor surface ofan imaging unit 21 included in the image sensor 13.

The image sensor 13 includes a stack of a semiconductor chip forming theimaging unit 21 and a semiconductor chip forming a logic unit 22, and aninterface for importing the output from the OIS driver 15 is mounted.

The imaging unit 21 captures the image of the subject formed by the lensunit 12 collecting light from the subject on the sensor surface on whicha plurality of pixels is arranged in a matrix, and outputs an imageacquired by the capturing.

The logic unit 22 supplies, to the signal processing circuit 17, imagedata obtained by adding position information of the lens unit 12 andangular velocity data output from the OIS driver 15 to the imagecaptured by the imaging unit 21 together with timing information whichindicates the timing for synchronizing the position information and theangular velocity data with coordinates on the image.

Specifically, the logic unit 22 receives the angular velocity datadetected by the motion sensor 14 and the position information of thelens unit 12 subjected to drive by the OIS actuator 16 at apredetermined sampling frequency (for example, 1 kHz) from the OISdriver 15. The logic unit 22 then adds, to the image data, the positioninformation of the lens unit 12 and the angular velocity data as well asan H-line counter of the image data at the timing the positioninformation and the angular velocity data are received, therebyoutputting the outcome. The position information of the lens unit 12,the angular velocity data, and the H-line counter may of course beoutput separately along with the image without being added to the image.The position information of the lens unit 12 and the angular velocitydata are associated line by line in the horizontal direction of theimage data as described above, whereby the signal processing circuit 17can synchronize the angular velocity data and the position informationwith the position in the vertical direction on the image. In otherwords, the H-line counter is used as the timing information forachieving the above synchronization.

Here, the H-line counter of the image data is, for example, a counterwhich is reset for each frame at a predetermined timing and incrementedby one each time one line in the horizontal direction is read, and isused for timing the position in the vertical direction on the image.Note that the H-line counter also counts in a blank section where noimage is read. Moreover, in addition to the H-line counter of the imagedata, time information such as a time stamp may be used as the timinginformation, for example. Note that a method of synchronizing theangular velocity data and the position information with the position inthe vertical direction on the image is described in detail in PatentDocument 2 above.

The motion sensor 14 physically (that is, not by image processing)detects motion of the imaging unit 21 and outputs informationrepresenting the motion. For example, in a case where the motion sensor14 includes a gyro sensor that can detect angular velocities in threeaxial directions, the motion sensor outputs angular velocity datarepresented by those angular velocities as information representing themotion of the imaging device 11.

Note that in addition to the sensor for OIS control, for example, asimple gyro sensor, a gyro sensor shared as one for OIS control (thatis, one having two ports), or the like can be used as the motion sensor14. Moreover, the motion sensor 14 is not limited to a gyro sensor butcan be a six-axis sensor or the like capable of outputting accelerationdata in addition to the angular velocity data in three axial directions.

On the basis of the angular velocity data output from the motion sensor14, the OIS driver 15 calculates an amount of movement by which the lensunit 12 is moved to optically cancel the occurrence of blurring in theimage captured by the imaging unit 21. Then, the OIS driver 15 suppliesthe amount of movement calculated to the OIS actuator 16, and performscontrol such that the lens unit 12 is disposed at a predeterminedposition according to the amount of movement. The OIS driver 15 furtheracquires the position information of the lens unit 12 driven by the OISactuator 16, and outputs the position information of the lens unit 12and the angular velocity data to the image sensor 13.

The OIS actuator 16 drives the lens unit 12 according to the amount ofmovement supplied from the OIS driver 15 to optically correct camerashake occurring in the image captured by the image sensor 13. Then, theOIS actuator 16 detects the position of the lens unit 12 being driven,and supplies the position information of the lens unit 12 to the OISdriver 15.

The signal processing circuit 17 is configured to perform correctionprocessing similar to that of the electronic image stabilizationproposed in Patent Document 2 while taking into consideration theposition information of the lens unit 12. That is, on the basis of theposition information of the lens unit 12 and the angular velocity datathat are added to the image data supplied from the image sensor 13, thesignal processing circuit 17 performs signal processing that correctsthe effect of the motion of the imaging unit 21 on the image (forexample, misalignment, edge deformation, distortion due to rollingshutter, deformation due to the effect of lens distortion, and the like)according to a correction function based on the position information ofthe lens unit 12 and the angular velocity data synchronized for eachcoordinate on the image. Note that the correction processing by thesignal processing circuit 17 will be described later with reference toFIG. 14.

The display 18 includes a display unit such as a liquid crystal panel oran organic electro luminescence (EL) panel, for example, and displays animage output from the signal processing circuit 17.

The recording medium 19 is a memory (for example, an electronicallyerasable and programmable read only memory (EEPROM)) that is a built-inmemory of the imaging device 11 or a removable memory detachable fromthe imaging device 11, and records an image output from the signalprocessing circuit 17.

In the imaging device 11 configured as described above, the signalprocessing circuit 17 can perform the correction processing according tothe electronic image stabilization on the image captured by the imagesensor 13 such that the occurrence of blurring is optically reduced. Asa result, as illustrated in FIG. 12, the imaging device 11 reduces theoccurrence of blurring within the exposure time to be able to morereliably correct image blurring (such as the misalignment, edgedeformation, and distortion due to rolling shutter caused by camerashake, the deformation due to the effect of lens distortion, and thelike).

Note that although the present embodiment describes the optical imagestabilizer of the barrel shift type in which the lens unit 12 is drivenby the OIS actuator 16, the imaging device 11 may adopt optical imagestabilizer of the sensor shift type in which the image sensor 13 isdriven by the OIS actuator 16. In this case, the OIS actuator 16supplies position information of the image sensor 13 instead of theposition information of the lens unit 12 to the OIS driver 15.

Moreover, the imaging device 11 of FIG. 13 is configured such that theangular velocity data output from the motion sensor 14 is supplied tothe image sensor 13 via the OIS driver 15. On the other hand, forexample, the imaging device 11 may be configured such that the motionsensor 14 includes two output ports used for outputting angular velocitydata and supplies the angular velocity data to each of the image sensor13 and the OIS driver 15. In this case, the angular velocity data is notsupplied from the OIS driver 15 to the image sensor 13.

Alternatively, for example, the imaging device 11 may include two of themotion sensors 14, in which case the two motion sensors 14 each supplythe angular velocity data to a corresponding one of the image sensor 13and the OIS driver 15. Moreover, in this case as well, the angularvelocity data is not supplied from the OIS driver 15 to the image sensor13.

Furthermore, although the image sensor 13 and the signal processingcircuit 17 are illustrated in different blocks in the imaging device 11of FIG. 13, the imaging device may adopt a configuration in whichprocessing by the signal processing circuit 17 is performed inside theimage sensor 13, for example. That is, the image sensor 13 can have astack structure in which a semiconductor chip forming the signalprocessing circuit 17 is stacked.

The correction processing according to the electronic imagestabilization performed by the signal processing circuit 17 will bedescribed with reference to FIG. 14.

As illustrated in FIG. 14, the optical center of an output image of theimage sensor 13 is set at a position O (0, 0). It is then assumed thatthe image sensor 13 rotates by an angle of rotation of −θp (rad) in thepitch direction, an angle of rotation of −θy (rad) in the yaw direction,and an angle of rotation of −θr (rad) in the roll direction. As a resultof such rotation, a point p (x, y) imaged on the image sensor 13 isassumed to be moved to a point P (X, Y).

Moreover, letting a point p0 (x0, y0) be the coordinates of a point atwhich distortion at the point p (x, y) is corrected and a point P0 (X0,Y0) be the coordinates of a point at which distortion at the point P (X,Y) is corrected, an image stabilization relational expression disclosedin Patent Document 1 above, that is, the following expression (1), holdstrue.

$\begin{matrix}{\mspace{79mu} \left\lbrack {{Expression}\mspace{14mu} 1} \right\rbrack} \\{\mspace{779mu} (1)} \\\left\{ \begin{matrix}{{X\; 0} = {{L \cdot \left( {\tan \left( {\alpha + \theta_{y}} \right)} \right)} + {x\; {0 \cdot \frac{\cos \mspace{11mu} \beta}{\cos \left( {\beta + \theta_{p}} \right)}}} + {x\; {0 \cdot \cos}\mspace{11mu} \theta_{r}} - {y\; {0 \cdot \sin}\; \theta_{r}} - {{2 \cdot x}\; 0}}} \\{{Y\; 0} = {{L \cdot \left( {\tan \left( {\beta + \theta_{p}} \right)} \right)} + {y\; {0 \cdot \frac{\cos \mspace{11mu} \alpha}{\cos \left( {\alpha + \theta_{y}} \right)}}} + {x\; {0 \cdot \sin}\mspace{11mu} \theta_{r}} + {y\; {0 \cdot \cos}\; \theta_{r}} - {{2 \cdot y}\; 0}}}\end{matrix} \right. \\{\mspace{79mu} {{\tan \mspace{11mu} \alpha} = {{\frac{x\; 0}{L}\mspace{14mu} \tan \mspace{11mu} \beta} = \frac{y\; 0}{L}}}}\end{matrix}$

Note that in expression (1), a focal length L is obtained by convertingthe focal length at the optical center position of the image sensor 13into the number of pixels, and is a value satisfying the followingexpression (2) using an amount of movement d of the position O (0, 0) ofthe optical center when the image sensor rotates by the angle ofrotation of −θ in the pitch direction or the yaw direction.

[Expression 2]

d=L·tan θ  (2)

Moreover, letting a function T be the image stabilization relationalexpression of the above expression (1), that is, the calculation forfinding the point P0 (X0, Y0) from the point p0 (x0, y0), the point P0(X0, Y0) is expressed by the following expression (3).

[Expression 3]

P0(X0,Y0)=T(x0,y0,L,θ _(p),θ_(y),θ_(r))  (3)

Furthermore, letting a function U be the calculation for finding thepoint p0 (x0, y0) from the point p (x, y), that is, the calculation forfinding at which position a point on an image subjected to the effect oflens distortion is located in a case where lens distortion is absent,the point p0 (x0, y0) is expressed by the following expression (4).

[Expression 4]

p0(x0,y0)=U(x,y)  (4)

Furthermore, letting a function D be the calculation for finding thepoint P (X, Y) from the point P0 (X0, Y0), that is, the calculation forfinding at which position on an image subjected to the effect of lensdistortion a point on an image in the absence of lens distortion islocated, the point P (X, Y) is expressed by the following expression(5).

[Expression 5]

P(X,Y)=D(X0,Y0)  (5)

Then, in a case where the signal processing circuit 17 performs thecorrection processing to output the result in which lens distortion iscorrected as illustrated in FIG. 8 above, for example, the point p0 (x0,y0) may be regarded as the point on the output image. That is, by usinga pixel value at the point P (X, Y) as a pixel value at the point p0(x0, y0) for each point in the output image, one can obtain an image onwhich image stabilization is performed and in which lens distortion iscorrected.

At this time, the point P (X, Y) can be found from the point p0 (x0, y0)by using the function T of expression (3) and the function D ofexpression (5) described above. That is, the point P0 (X0, Y0) can befound from the point p0 (x0, y0) using the function T of expression (3),and the point P (X, Y) can be found from the point P0 (X0, Y0) using thefunction D of expression (5). Here, letting a function F be a compositefunction of the function T and the function D, the point P (X, Y) isexpressed by the following expression (6).

[Expression 6]

P(X,Y)=F(x0,y0,L,θ _(p),θ_(y),θ_(r))  (6)

On the other hand, in a case where the signal processing circuit 17performs the correction processing to output the result in which lensdistortion is not corrected as illustrated in FIG. 7 above, for example,the point p (x, y) may be regarded as the point on the output image.That is, by using the pixel value at the point P (X, Y) as a pixel valueat the point p (x, y) for each point in the output image, one can obtainan image on which image stabilization is performed and in which lensdistortion is not corrected.

At this time, the point P (X, Y) can be found from the point p (x, y) byusing the function T of expression (3), the function U of expression(4), and the function D of expression (5) described above. That is, thepoint p0 (x0, y0) can be found from the point p (x, y) using thefunction U of expression (4), the point P0 (X0, Y0) can be found fromthe point p0 (x0, y0) using the function T of expression (3), and thepoint P (X, Y) can be found from the point P0 (X0, Y0) using thefunction D of expression (5). Here, letting a function G be a compositefunction of the function T, the function U, and the function D, thepoint P (X, Y) is expressed by the following expression (7).

[Expression 7]

P(X,Y)=G(x,y,L,θ _(p),θ_(y),θ_(r))  (7)

Note that the coordinate values of the point P (X, Y) found byexpressions (6) and (7) rarely take integer values, but the pixel valuein the output image can be calculated by interpolation from a pixelvalue of neighboring coordinates. Moreover, the pixel value at eachpoint in the output image can be found by calculating a correspondingcoordinate position on the input image using the above functions foreach point. In addition, for example, the pixel value may be calculatedby dividing the output image and calculating a corresponding coordinateposition on the input image using the above functions only for gridpoints, and finding a coordinate position by interpolation for pointsother than the grid points.

Note that the description is herein made about calculating the pixelvalue at a certain timing to explain the principle briefly, but inpractice, the imaging time of pixels within one screen varies dependingon the pixel positions. Accordingly, the pixel value of each pixel iscalculated by using the angle of rotation −θp (rad) in the pitchdirection, the angle of rotation −θy (rad) in the yaw direction, and theangle of rotation −θr (rad) in the roll direction corresponding to thepixel position.

Now, the correction processing according to optical image stabilizer andelectronic image stabilization is implemented by adding the amount ofmovement by which the OIS actuator 16 moves the lens unit 12 to thecorrection processing based on the function F of expression (6) and thefunction G of expression (7) above. Although the correction processingusing the function F of expression (6) will be described below, thecorrection processing using the function G of expression (7) can also beexecuted in a manner similar to that of the correction processing usingthe function F of expression (6).

First, the point P0 (X0, Y0) is set as the coordinates on the inputimage (image with camera shake in a case where optical image stabilizeris not performed) corresponding to the coordinates of the point p0 (x0,y0) on the output image that is subjected to the correction processingaccording to optical image stabilizer and electronic imagestabilization. At this time, as described above, the function F forcalculating the coordinates according to electronic image stabilizationis expressed by expression (6).

Moreover, the correction processing according to optical imagestabilizer (employing the barrel shift method or sensor shift method)may be regarded as parallel movement of an image. Accordingly,coordinates (X_(ois), Y_(ois)) on the input image corresponding to thecoordinates p0 (x0, y0) on the output image are found by the followingexpression (8) using a shift amount (x_(ois), y_(ois)) according tooptical image stabilizer for each pixel.

$\begin{matrix}\left\lbrack {{Expression}\mspace{14mu} 8} \right\rbrack & \; \\\begin{matrix}{\left( {X_{ois},Y_{ois}} \right) = {\left( {X,Y} \right) - \left( {x_{ois},y_{ois}} \right)}} \\{= {{F\left( {{x\; 0},{y\; 0},L,\theta_{p},\theta_{y},\theta_{r}} \right)} - \left( {X_{ois},Y_{ois}} \right)}}\end{matrix} & (8)\end{matrix}$

As a result, an image subjected to the correction processing accordingto optical image stabilizer and electronic image stabilization can beoutput by outputting the pixel value of the coordinates (X_(ois),Y_(ois)) on the input image as the pixel value of the coordinates (x0,y0) on the output image.

Note that the coordinate values of the coordinates (X_(ois), Y_(ois))found by expressions (8) rarely take integer values, but the pixel valuein the output image can be calculated by interpolation from a pixelvalue of neighboring coordinates. Moreover, the pixel value at eachpoint in the output image can be found by calculating a correspondingcoordinate position on the input image using the above functions foreach point. In addition, for example, the pixel value may be calculatedby dividing the output image and calculating a corresponding coordinateposition on the input image using the above functions only for gridpoints, and finding a coordinate position by interpolation for pointsother than the grid points.

Note that the description is herein made about calculating the pixelvalue at a certain timing to explain the principle briefly, but inpractice, the imaging time of pixels within one screen varies dependingon the pixel positions. Accordingly, the pixel value of each pixel iscalculated by using the angle of rotation −θp (rad) in the pitchdirection, the angle of rotation −θy (rad) in the yaw direction, and theangle of rotation −θr (rad) in the roll direction corresponding to thepixel position as well as the shift amount according to optical imagestabilizer.

Here, a description will be given of a case where Hall data obtained byreading the position of the lens unit 12 using a Hall element is used asthe position information of the lens unit 12 driven in optical imagestabilizer. For example, the logic unit 22 can add the angular velocitydata detected by the motion sensor 14 and the Hall data obtained byreading the position of the lens unit 12 to the image data together withthe H-line counter of the image data, and output the image data to whichthe above data is added. At this time, the logic unit 22 performs timingadjustment to synchronize the timing at which the motion sensor 14detects the angular velocity data and the timing at which the Hallelement reads the position of the lens unit 12 on the basis of a delaytime up to the timing of acquisition of the angular velocity data andthe Hall data, the relationship of exposure end time and exposure timefor each pixel (H line), and the like.

In this case, the coordinates (x0, y0) on the output image (image afterimage stabilization) are found by the following expression (9) using theshift amount (x_(ois), y_(ois)) according to optical image stabilizer,Hall data values (hx, hy), Hall data (hx0, hy0) when the lens unit 12 isat the center in optical image stabilizer, and pixel count conversionfactors (kx, ky).

[Expression 9]

(x _(ois) ,y _(ois))=(kx·(hx−hx0),ky·(hy−hy0))  (9)

Then, by inputting the shift amount (x_(ois), y_(ois)) found byexpression (9) to the above expression (8), the coordinates (X_(ois),Y_(ois)) on the input image (OIS output image) corresponding to thecoordinates p0 (x0, y0) on the output image (image after imagestabilization) are determined. As a result, an image on which imagestabilization is performed can be created by using the pixel value ofthe coordinates. Here, expression (9) illustrates an example in whichthe conversion processing is performed assuming that the amount ofchange in the Hall data (hx0, hy0) and the amount of movement of thepixel position are in a linear relationship. On the other hand, in acase where they are not in a linear relationship, for example, theconversion processing is performed in accordance with the relationshipbetween the amount of change in the Hall data (hx0, hy0) and the amountof movement of the pixel position.

Note that the pixel value at each point in the output image can be foundby calculating a corresponding coordinate position on the input imageusing the above functions for each point. In addition, for example, thepixel value may be calculated by dividing the output image andcalculating a corresponding coordinate position on the input image usingthe above functions only for grid points, and finding a coordinateposition by interpolation for points other than the grid points.

Note that the description is herein made about calculating the pixelvalue at a certain timing to explain the principle briefly, but inpractice, the imaging time of pixels within one screen varies dependingon the pixel positions. Accordingly, the pixel value of each pixel iscalculated by using the angle of rotation −θp (rad) in the pitchdirection, the angle of rotation −θy (rad) in the yaw direction, and theangle of rotation −θr (rad) in the roll direction corresponding to thepixel position as well as the shift amount (Hall data values (hx, hy))according to optical image stabilizer.

<Image Stabilization Processing of Imaging Device>

An example of image stabilization processing executed in an imagingmethod by the imaging device 11 will be described with reference to aflowchart in FIG. 15.

In the imaging device 11, for example, the image stabilizationprocessing is started when the imaging unit 21 starts imaging of oneframe, and in step S11, the OIS driver 15 acquires angular velocity dataoutput from the motion sensor 14.

In step S12, the OIS driver 15 calculates an amount of movement by whichthe lens unit 12 is moved on the basis of the angular velocity dataacquired in step S11, and supplies the amount of movement to the OISactuator 16.

In step S13, the OIS actuator 16 performs optical image stabilization bydriving the lens unit 12 according to the amount of movement suppliedfrom the OIS driver 15 in step S12.

In step S14, the OIS actuator 16 detects the position of the lens unit12 driven in step S13, and supplies position information of the lensunit 12 to the OIS driver 15. The OIS driver 15 then supplies theposition information of the lens unit 12 and the angular velocity dataacquired in step S11 to the logic unit 22 of the image sensor 13.

In step S15, the logic unit 22 adds the position information of the lensunit 12 and the angular velocity data supplied from the OIS driver 15 instep S14 to the image data output from the imaging unit 21 together withan H-line counter of the image data corresponding to the timing ofreception of the position information and the angular velocity data, andsupplies the image data to which the information and data are added tothe signal processing circuit 17.

In step S16, the signal processing circuit 17 uses the positioninformation of the lens unit 12 and the angular velocity data to performelectronic image stabilization processing on the image data supplied instep S15 according to a function that converts the position for eachcoordinate on the image data synchronized with the position informationand the angular velocity data. The processing is thereafter ended, andsimilar processing is repeated each time the imaging unit 21 startsimaging of a next frame. Note that the correction processing isperformed continuously without being ended in shooting of a video or thelike, on a preview screen, in continuous shooting of still images, andthe like in which image stabilization is performed continuously.Moreover, the processing from step S11 to step S14 is performedcontinuously at a preset sampling frequency.

As described above, the imaging device 11 can reliably correct blurringby reducing the occurrence of blurring within the exposure time by theoptical image stabilization according to the control of the OIS driver15, and by reducing the effect of camera shake on the image by theelectronic image stabilization processing performed by the signalprocessing circuit 17.

A result of correction of an image captured by the imaging device 11employing such an imaging method will be described with reference toFIG. 16.

It is assumed, for example, that the optical image stabilizer canperform correction by the angle of ±1.5 degrees while the electronicimage stabilization can perform correction by the angle of ±6 degrees.At this time, with respect to vibration as illustrated in FIG. 16, theoptical image stabilizer (OIS) corrects only a high frequency componentto be able to produce a correction result that blurring within theexposure time is reduced. Then, the optical image stabilizer (OIS) andthe electronic image stabilization (EIS) correct a low frequencycomponent to be able to produce a correction result that the angle iskept at substantially zero degree.

As described above, the imaging device 11 performs imaging whileperforming the correction processing according to the optical imagestabilizer, and can perform the electronic image stabilization on theimage being captured by using the position information of the lens unit12 (information of the optical image stabilizer) and the angularvelocity data. The imaging device 11 can thus perform imagestabilization while accommodating a difference in the amount of movementdue to the position within an image plane and a difference in the amountof movement due to a gap in the exposure timing within one screen.

Therefore, the imaging device 11 can accurately correct camera shakefrom the center to the edge by correcting the effects of the edgedeformation, lens distortion, and rolling shutter while reducing theoccurrence of blurring within the exposure time not only in the imagingperformed with a short exposure time such as in the daytime outdoors,but also in the imaging performed with a long exposure time such as in adark place, or at night.

Moreover, it is typically difficult to increase the range of correctionof the optical image stabilizer since the device needs to be increasedin size, or large power is required for control in order to increase therange of correction. On the other hand, the imaging device 11 canperform correction on a wider range by using the electronic imagestabilization to cover the range that cannot be corrected by the opticalimage stabilizer. Furthermore, while it is difficult for the opticalimage stabilizer to accommodate correction in the rotation direction,the imaging device 11 can perform correction in the rotation direction.

<Second Embodiment of Imaging Device>

FIG. 17 is a block diagram illustrating an example of the configurationof a second embodiment of an imaging device to which the presenttechnology is applied. Note that in an imaging device 11A illustrated inFIG. 17, a configuration common to that of the imaging device 11 in FIG.13 will be assigned the same reference numeral as that assigned theretoand will not be described in detail.

As illustrated in FIG. 17, the imaging device 11A includes the lens unit12, the motion sensor 14, the OIS actuator 16, the signal processingcircuit 17, the display 18, the recording medium 19, and the imagingunit 21 as with the imaging device 11 of FIG. 13.

Then in the imaging device 11A, a logic unit 22A of an image sensor 13Aand an OIS driver 15A have configurations different from those of theimaging device 11 in FIG. 13.

The logic unit 22A generates OIS control information instructingexecution or termination of optical image stabilizer according to theexposure timing at which the imaging unit 21 performs exposure, andsupplies the OIS control information to the OIS driver 15A. Suchprocessing of generating the OIS control information according to theexposure timing of the imaging unit 21 is preferably performed in thelogic unit 22A incorporated in the image sensor 13A.

The logic unit 22A generates the OIS control information on the basis ofan exposure end (read end) timing and an exposure start timing of thenext frame of the imaging unit 21, for example. The logic unit 22A canalso specify the exposure start timing of the next frame on the basis ofinformation such as the time between frames, the exposure time of thenext frame (which changes depending on imaging conditions due to anautomatic exposure function or the like). Those timings are determinedand operated inside the image sensor 13A, so that the OIS controlinformation can be generated in the logic unit 22A more easily than inthe configuration in which the OIS control information is generatedoutside the image sensor 13A.

On the basis of the OIS control information supplied from the logic unit22A, the OIS driver 15A performs an operation to bring the lens unit 12back to the center position in a case where the OIS control informationinstructs termination of the optical image stabilizer.

Alternatively, in a case where the OIS control information is switchedto one instructing execution of the optical image stabilizer while thelens unit 12 is not completely returned to the center position in themiddle of the operation by the OIS driver 15A bringing the lens unit 12back to the center position in accordance with the OIS controlinformation instructing termination of the optical image stabilizer, theOIS driver can perform the optical image stabilizer from the position ofthe lens unit 12 on the way back to the center position.

The imaging device 11A configured as described above can perform there-centering processing of the optical image stabilizer between framesby the logic unit 22A supplying the OIS control information to the OISdriver 15A. As a result, the imaging device 11A can perform the opticalimage stabilizer while resetting the lens position between frames,thereby being able to perform correction using the entire range ofangles by which the optical image stabilizer can perform correction atall times in each frame.

That is, in a case where vibration occurs with the amplitude exceedingthe angle by which the optical image stabilizer can perform correction,the imaging device 11 of FIG. 13 cannot reduce blurring within theexposure time during the vibration in the excessive range. On the otherhand, the imaging device 11A performs the re-centering processing of theoptical image stabilizer to be able to reduce the occurrence of blurringwithin the exposure time even on the occurrence of vibration with alarge amplitude, if the vibration within one frame is within the angleby which the optical image stabilizer can perform correction.

The OIS control information generated by the logic unit 22A will bedescribed with reference to FIG. 18.

Note that the horizontal axis of the graphs illustrated in FIGS. 18 to20 represents time and illustrates a change with time. Moreover, aparallelogram in the figure schematically represents the time taken forthe image data to be read while an exposure is performed from the top tothe bottom of an image (which may be from the bottom to the topdepending on the setting of imaging) at the time the image is capturedby a CMOS image sensor. In the illustrated example, an electronicshutter is opened in order from the top of the image, and reading isperformed sequentially from the top after the exposure is made for acertain time.

As illustrated in A of FIG. 18, in a case where there is a time duringwhich the exposure does not overlap between frames after the end ofreading of the bottom of the image and before the opening of theelectronic shutter at the top of the image of the next frame, the logicunit 22A outputs the OIS control information (OIS enable) instructingexecution of the optical image stabilizer during the period in which theexposure is performed. Moreover, the logic unit 22A outputs the OIScontrol information (OIS disable) instructing termination of the opticalimage stabilizer during the period in which the exposure is notperformed. For example, the logic unit 22A outputs the OIS controlinformation (OIS disable) instructing termination of the optical imagestabilizer in a case where the time from the end of the exposure to thestart of the next exposure is longer than or equal to a predeterminedtime set.

Note that, in consideration of a delay in actual control, the logic unit22A can shift the timing for switching between execution and terminationof the optical image stabilizer by offset times (a first offset and asecond offset illustrated in FIG. 18) set for the read end timing andthe exposure start timing, respectively.

On the other hand, as illustrated in B of FIG. 18, the logic unit 22Aconstantly outputs the OIS control information (OIS enable) instructingexecution of the optical image stabilizer in the absence of a periodduring which the exposure does not overlap between frames or in a casewhere the period during which the exposure does not overlap betweenframes is shorter than a predetermined time set. That is, in the casewhere the exposure always overlaps between frames, the optical imagestabilizer is performed continuously without the re-centering processingof the optical image stabilizer.

Processing of returning a control position of the optical imagestabilizer to the center between frames will be described with referenceto FIG. 19.

Illustrated in A of FIG. 19 is an example in which the control positionof the optical image stabilizer is returned to the center immediatelybefore the exposure is started.

In the case where the control position of the optical image stabilizeris returned to the center immediately before the exposure is started asin the figure, the optical image stabilizer can be performed by movingthe lens unit 12 from the center position. Note that the effect ofvibration appears on the image in a case where the exposure time islonger than the example illustrated in A of FIG. 19 and the exposure ofthe next frame is started while the control of the optical imagestabilizer is not stable. In order to avoid this, the predetermined timecan be set such that the OIS control information (OIS enable)instructing execution of the optical image stabilizer is constantlyoutput in the case where the period during which the exposure does notoverlap between frames is shorter than the predetermined time set.

On the other hand, as illustrated in B of FIG. 19, in a case where theexposure time is shorter than the example illustrated in A of FIG. 19,the effect of vibration does not appear on the image so that the controlposition of the optical image stabilizer can be reliably returned to thecenter.

Incidentally, there occurs a period during which control is disabled inthe presence of hunting at the time the lens unit 12 is returned to thecenter position as illustrated in A and B of FIG. 19. It is thuspreferable to execute the re-centering processing of the optical imagestabilizer such that the hunting does not occur at the time the lensunit 12 is returned to the center position.

In the case where the lens unit 12 can be reset to the center positionbetween frames as described above, the range of correction of theoptical image stabilizer can be secured widely at all times. On theother hand, in a case where there is not enough time to reset the lensunit 12 to the center position between frames, the re-centeringprocessing of the optical image stabilizer cannot be executed. Moreover,the re-centering processing of the optical image stabilizer cannot beexecuted either in a case where the period during which the exposuredoes not overlap between frames is not always longer than or equal tothe time for the lens unit 12 to return to the center position and bestable from the position corresponding to the maximum amount ofmovement.

Accordingly, instead of always returning the lens unit 12 to the centerposition, there can be adopted processing of returning the lens unit 12toward the center, that is, processing of performing the optical imagestabilizer in the middle of the operation that brings the lens unit 12back to the center position.

The processing of returning the control position of the optical imagestabilizer toward the center between frames will be described withreference to FIG. 20.

As illustrated in A of FIG. 20, the control is performed such thathunting does not occur at the time the lens unit 12 is returned to thecenter position. Moreover, A of FIG. 20 illustrates an example in whichthe control position of the optical image stabilizer is returned to thecenter before the exposure is started, as with A and B of FIG. 19.

On the other hand, as illustrated in B of FIG. 20, even if the controlposition of the optical image stabilizer is not completely returned tothe center when the exposure is started, the correction processingaccording to the optical image stabilizer is performed from the positionmidway through the returning. Although the movement of the lens unit 12is not started from the center, such control enables execution of thecorrection processing according to the optical image stabilizer duringthe exposure period so that there is no adverse effect on the image.

As described above, even in the case where the period during which theexposure does not overlap between frames is shorter than the timerequired to return the lens unit 12 to the center, the correctable rangein the next frame can be expanded as much as possible by returning thelens unit 12 toward the center even only to some extent. That is, inthis case, the control to return the lens unit 12 toward the center isperformed to be able to increase the amount of movement that can be usedfor the optical image stabilizer in the next frame compared to when suchcontrol is not performed, whereby the occurrence of blurring within theexposure time can be reduced as much as possible.

Note that in a case where vibration occurs with the amplitude exceedingthe angle by which the optical image stabilizer can perform correctionduring the exposure in the frame, the electronic image stabilization canbe performed effectively to be able to correct blurring in the imageeven if the blurring within the exposure time cannot be reducedsufficiently.

Moreover, the imaging device 11A performs signal processing thatperforms correction on the basis of the angular velocity data outputfrom the motion sensor 14 and the position information of the lens unit12 for each coordinate on the image. Therefore, the signal processingcircuit 17 can perform processing using the same algorithm for any of acase where the lens unit 12 is returned to the center, a case where thelens unit 12 is not completely returned to the center, a case whereregular optical image stabilizer is applied, and a case where the lensunit 12 is always fixed at the center position, for example.

<Application to Still Image>

The imaging device 11 can be effectively used not only for imagestabilization processing at the time of shooting a video but also forcamera shake at the time of capturing a still image, where there is alsoa method for improving the performance of reducing camera shake in astill image.

For example, when a still image is captured in a dark environment suchas in a dark place or at night, the exposure time is decreased so thatthe amount of light becomes insufficient and that a noisy image iscaptured even if imaging is performed with high gain. A long exposuretime needs to be set in order to reduce the occurrence of such noise, inwhich case the image becomes susceptible to camera shake and hasblurring in many cases.

Accordingly, the optical image stabilizer is used to reduce blurringthat occurs during exposure to be able to reduce blurring in a stillimage captured with long exposure. However, the optical image stabilizercannot correct the effect of lens distortion or the edge deformation asdescribed above, so that the image is captured with blurring occurringon the edge portion though no blurring occurs in the center of thescreen. Moreover, blurring on the screen cannot be reduced in a casewhere vibration occurs with the amplitude exceeding the range ofcorrection of the optical image stabilizer during exposure.

On the other hand, the imaging device 11 can be effectively applied notonly to image stabilization processing at the time of shooting a videobut also to camera shake at the time of capturing a still image, and canreliably correct blurring from the center to the edge of the image incapturing a still image that requires long exposure.

Blurring that occurs in a still image will be described with referenceto FIG. 21.

Each of B and C of FIG. 21 illustrates a still image that is capturedwhen exposure is performed with a cycle A under the vibration conditionsas illustrated in A of FIG. 21. Illustrated in B of FIG. 21 is a stillimage output without being subjected to the correction processing thatcorrects blurring related to vibration, while C of FIG. 21 illustrates astill image subjected to the correction processing according to theoptical image stabilizer.

As illustrated in C of FIG. 21, blurring can be reduced by performingthe correction processing according to the optical image stabilizer butremains on the edge portion of the still image due to the effect of edgedeformation.

A still image on which the correction processing is performed accordingto the image stabilization proposed in Patent Document 2 will bedescribed with reference to FIG. 22.

Illustrated in A of FIG. 22 is an example of an image obtained bycorrecting the lens distortion itself when the correction processingaccording to the image stabilization in Patent Document 2 is performed,the image being obtained by superimposing eight sequential images thatare each captured sequentially with the exposure time one-eighth of thatof normal exposure.

Illustrated in B of FIG. 22 is an example of an image obtained withoutcorrecting the lens distortion itself when the correction processingaccording to the image stabilization in Patent Document 2 is performed,the image being obtained by superimposing eight sequential images thatare each captured sequentially with the exposure time one-eighth of thatof normal exposure.

As illustrated in A and B of FIG. 22, the correction processingaccording to the image stabilization in Patent Document 2 is performedto be able to superimpose the sequential images with almost nomisalignment from the center to the edge of the image. That is, theimage in which the occurrence of blurring is reduced and in whichblurring within the exposure time is reduced by the decrease in theexposure time can be obtained by performing the correction processingaccording to the image stabilization in Patent Document 2 andsuperimposing a plurality of sequential images sequentially capturedwith a short exposure time. Note that although an increase in theoccurrence of noise is a concern in the case where the exposure time isdecreased, the noise can be reduced by superimposing a plurality ofimages.

However, the correction processing according to the image stabilizationin Patent Document 2 alone allows the effect of blurring within theexposure time to remain on the screen as a whole, so that the exposuretime needs to be decreased further in order to make blurring within theexposure time less noticeable.

A still image on which the correction processing is performed accordingto the image stabilization by the imaging device 11 in FIG. 13 will bedescribed with reference to FIG. 23.

Illustrated in A of FIG. 23 is an example of an image obtained bycorrecting the lens distortion itself when the correction processingaccording to the image stabilization by the imaging device 11 isperformed, the image being obtained by superimposing eight sequentialimages that are each captured sequentially with the exposure timeone-eighth of that of normal exposure.

Illustrated in B of FIG. 23 is an example of an image obtained withoutcorrecting the lens distortion itself when the correction processingaccording to the image stabilization by the imaging device 11 isperformed, the image being obtained by superimposing eight sequentialimages that are each captured sequentially with the exposure timeone-eighth of that of normal exposure.

In the imaging device 11, as described above, the signal processingcircuit 17 performs the correction processing according to theelectronic image stabilization on the image captured by the image sensor13 such that the occurrence of blurring is optically reduced.

As illustrated in A and B of FIG. 23, the sequential images can besuperimposed with absolutely no occurrence of blurring from the centerto the edge of the image upon reducing the occurrence of blurring withinthe exposure time by performing the correction processing according tothe image stabilization by the imaging device 11 and superimposing aplurality of sequential images sequentially captured with a shortexposure time.

Blurring that occurs in a still image by larger vibration will bedescribed with reference to FIG. 24.

Illustrated in each of B and C of FIG. 24 is a still image that iscaptured when exposure is performed with the cycle A under the vibrationconditions exceeding the range of correction of the optical imagestabilizer as illustrated in A of FIG. 24, the range of correction beingset to ±1.5 degrees. Illustrated in B of FIG. 24 is a still image outputwithout being subjected to the correction processing that correctsblurring related to vibration, while C of FIG. 24 illustrates a stillimage subjected to the correction processing according to the opticalimage stabilizer.

As illustrated in A of FIG. 24, the oscillatory wave in the yawdirection is so large as to exceed the range of correction (±1.5degrees) of the optical image stabilizer, whereby the correctionprocessing according to the optical image stabilizer cannot reduceblurring that occurs in the still image as illustrated in C of FIG. 24.

A still image on which the correction processing is performed againstthe vibration illustrated in FIG. 24 by the image stabilization proposedin Patent Document 2 will be described with reference to FIG. 25.

Illustrated in A of FIG. 25 is an example of an image obtained bycorrecting the lens distortion itself when the correction processingaccording to the image stabilization in Patent Document 2 is performed,the image being obtained by superimposing eight sequential images thatare captured sequentially each with the exposure time one-eighth of thatof normal exposure.

Illustrated in B of FIG. 25 is an example of an image obtained withoutcorrecting the lens distortion itself when the correction processingaccording to the image stabilization in Patent Document 2 is performed,the image being obtained by superimposing eight sequential images thatare captured sequentially each with the exposure time one-eighth of thatof normal exposure.

As illustrated in A and B of FIG. 25, the correction processingaccording to the image stabilization in Patent Document 2 is performedto be able to superimpose the sequential images with almost nomisalignment from the center to the edge of the image. However, theimage is blurred as a whole since the effect of blurring within theexposure time remains.

A still image on which the correction processing is performed againstthe vibration illustrated in A of FIG. 24 by the image stabilization ofthe imaging device 11 in FIG. 13 will be described with reference toFIG. 26.

Illustrated in A of FIG. 26 is an example of an image obtained bycorrecting the lens distortion itself when the correction processingaccording to the image stabilization of the imaging device 11 isperformed, the image being obtained by superimposing eight sequentialimages that are captured sequentially each with the exposure timeone-eighth of that of normal exposure.

Illustrated in B of FIG. 26 is an example of an image obtained withoutcorrecting the lens distortion itself when the correction processingaccording to the image stabilization of the imaging device 11 isperformed, the image being obtained by superimposing eight sequentialimages that are captured sequentially each with the exposure timeone-eighth of that of normal exposure.

In the imaging device 11, as described above, the signal processingcircuit 17 performs the correction processing according to theelectronic image stabilization on the image captured by the image sensor13 such that the occurrence of blurring is optically reduced.

As illustrated in A and B of FIG. 26, the sequential images can besuperimposed without misalignment from the center to the edge of theimage upon reducing the occurrence of blurring within the exposure timeby performing the correction processing according to the imagestabilization by the imaging device 11 and superimposing a plurality ofsequential images sequentially captured with a short exposure time.However, the effect of blurring within the exposure time remains by theamount by which the amplitude of the vibration occurring exceeds therange of correction of the optical image stabilization, whereby theimage is blurred as a whole though less blurry than the example of FIG.25.

A still image on which the correction processing is performed againstthe vibration illustrated in A of FIG. 24 by the image stabilization ofthe imaging device 11A in FIG. 17 will be described with reference toFIG. 27.

Illustrated in A of FIG. 27 is an example of an image obtained bycorrecting the lens distortion itself when the correction processingaccording to the image stabilization by the imaging device 11A isperformed, the image being obtained by superimposing eight sequentialimages that are captured sequentially each with the exposure timeone-eighth of that of normal exposure.

Illustrated in B of FIG. 27 is an example of an image obtained withoutcorrecting the lens distortion itself when the correction processingaccording to the image stabilization by the imaging device 11A isperformed, the image being obtained by superimposing eight sequentialimages that are captured sequentially each with the exposure timeone-eighth of that of normal exposure.

In the imaging device 11A, as described above, the signal processingcircuit 17 performs the correction processing according to theelectronic image stabilization on the image captured by the image sensor13 such that the occurrence of blurring is optically reduced while there-centering processing of the optical image stabilizer is performed.

As illustrated in A and B of FIG. 27, the sequential images can besuperimposed without misalignment from the center to the edge of theimage and at the same time the effect of blurring within the exposuretime can be substantially reduced by performing the correctionprocessing according to the image stabilization by the imaging device11A and superimposing a plurality of sequential images sequentiallycaptured with a short exposure time while performing the re-centeringprocessing of the optical image stabilizer.

As described with reference to FIGS. 21 to 27, the image stabilizationprocessing to which the present technology is applied is effective forfurther correcting blurring in the still image when combined with theprocessing that superimposes a plurality of sequential imagessequentially captured with a short exposure time.

Moreover, the imaging device 11 of the present embodiment does notrequire processing of comparing image data unlike a configuration thatuses an image subjected to the optical image stabilizer to detect amovement vector by comparing image data and perform the electronic imagestabilization, for example. The imaging device 11 can thus correctblurring with high accuracy by the processing having less load.

Furthermore, as compared to the processing disclosed in Patent Document1 described above, for example, the imaging device 11 of the presentembodiment performs correction on the basis of the angular velocity dataoutput from the motion sensor 14 and the position information of thelens unit 12 even in the occurrence of an error in the control of theoptical image stabilizer, thereby being able to avoid the image beingaffected by the error.

Note that each processing described with reference to the aforementionedflowchart need not be performed chronologically in the order listed inthe flowchart but includes processing executed concurrently orseparately (for example, parallel processing or processing performed byan object). Moreover, the program may be processed by a single CPU orprocessed in a distributed manner by a plurality of CPUs.

Furthermore, although the above embodiment describes the configurationof the imaging device 11, the present invention can also be applied to acamera module including at least the image sensor 13, the motion sensor14, the OIS driver 15, the OIS actuator 16, and the signal processingcircuit 17, or various electronic apparatuses equipped with the cameramodule.

Furthermore, the imaging device 11 need not include the logic unit 22performing the electronic image stabilization on the image output fromthe image sensor 13. That is, the function of the logic unit 22 may beincluded in a unit different from the imaging device 11 to allow outputof image data to which the position information of the lens unit 12 andthe angular velocity data in synchronization with the position on theimage are added. As a matter of course, the image stabilizationprocessing can be executed with high accuracy and the system can bebuilt with ease by adopting the configuration in which the imagingdevice 11 includes the logic unit 22, more preferably, the configurationin which the signal processing circuit 17 is included on thesemiconductor chip of the stacked image sensor 13.

Note that in the above embodiment, the camera shake occurring in theimaging device 11 (that is, the vibration of the image sensor 13incorporated in the imaging device 11) is defined by the rotation in thepitch direction, the yaw direction, and the roll direction asillustrated in FIG. 28.

<Example of Use of Image Sensor>

FIG. 29 is a diagram illustrating an example in which the aforementionedimage sensor is used.

The aforementioned image sensor can be used in various cases for sensinglight such as visible light, infrared light, ultraviolet light, orX-rays as described below, for example.

-   -   A device such as a digital camera or a portable device with a        camera function for photographing an image to be used for        viewing    -   A device for use in transportation such as an in-vehicle sensor        that photographs the front, back, periphery, interior, and the        like of a vehicle for safe driving such as automatic stop,        recognizing the condition of a driver, or the like, a        surveillance camera that monitors traveling vehicles and roads,        or a range sensor that measures the distance between vehicles        and the like    -   A device for use in home appliances such as a TV, a        refrigerator, or an air conditioner to photograph a gesture of a        user and operate an appliance in accordance with the gesture    -   A device for use in medical and health care such as an endoscope        or a device that performs angiography by receiving infrared        light    -   A device for use in security such as a surveillance camera used        for crime prevention or a camera used for person authentication    -   A device for use in beauty care such as a skin measuring        instrument that photographs skin or a microscope that        photographs the scalp    -   A device for use in sports such as an action camera or a        wearable camera for sports applications and the like    -   A device for use in agriculture such as a camera that monitors        the condition of fields and crops

<Example of Application to Endoscopic Surgical System>

The technology according to the present disclosure (the presenttechnology) can be applied to various products. For example, thetechnology according to the present disclosure may be applied to anendoscopic surgical system.

FIG. 30 is a diagram illustrating an example of a schematicconfiguration of an endoscopic surgical system to which the technologyaccording to the present disclosure (the present technology) can beapplied.

FIG. 30 illustrates how a surgeon (doctor) 11131 performs surgery on apatient 11132 who is on a patient bed 11133 by using an endoscopicsurgical system 11000. As illustrated in the figure, the endoscopicsurgical system 11000 includes an endoscope 11100, other surgicalinstruments 11110 such as a pneumoperitoneum tube 11111 and an energytreatment tool 11112, a support arm device 11120 for supporting theendoscope 11100, and a cart 11200 carrying various devices for anendoscopic surgery.

The endoscope 11100 includes a lens barrel 11101, a predetermined lengthof which from the tip is inserted into the body cavity of the patient11132, and a camera head 11102 connected to the proximal end of the lensbarrel 11101. Although the example in the figure illustrates theendoscope 11100 configured as a so-called rigid scope with the lensbarrel 11101 being rigid, the endoscope 11100 may be configured as aso-called flexible scope with a flexible lens barrel.

The tip of the lens barrel 11101 is provided with an opening to which anobjective lens is fitted. A light source device 11203 is connected tothe endoscope 11100 so that light generated by the light source device11203 is guided to the tip of the lens barrel by a light guide extendinginside the lens barrel 11101 and is projected toward an observationtarget in the body cavity of the patient 11132 via the objective lens.Note that the endoscope 11100 may be a direct-viewing endoscope, anoblique-viewing endoscope, or a side-viewing endoscope.

An optical system and an imaging element are provided inside the camerahead 11102, and reflected light (observation light) from the observationtarget is concentrated on the imaging element by the optical system. Theobservation light is photoelectrically converted by the imaging elementso that an electrical signal corresponding to the observation light,that is, an image signal corresponding to an observation image, isgenerated. The image signal is transmitted as RAW data to a cameracontrol unit (CCU) 11201.

The CCU 11201 includes a central processing unit (CPU), a graphicsprocessing unit (GPU), or the like and has control over the operationsof the endoscope 11100 and a display device 11202. Moreover, the CCU11201 receives the image signal from the camera head 11102 and performsvarious image processings such as development processing (demosaicprocessing) on the image signal for displaying an image based on theimage signal, for example.

Under the control of the CCU 11201, the display device 11202 displaysthe image based on the image signal that is subjected to the imageprocessing by the CCU 11201.

The light source device 11203 includes a light source such as a lightemitting diode (LED), for example, and supplies irradiation light forphotographing a surgical site or the like to the endoscope 11100.

An input device 11204 is an input interface for the endoscopic surgicalsystem 11000. A user can input various information and instructions tothe endoscopic surgical system 11000 via the input device 11204. Forexample, the user inputs an instruction or the like to change imagingconditions (the type of irradiation light, magnification, focal length,and the like) adopted by the endoscope 11100.

A treatment tool control device 11205 controls drive of the energytreatment tool 11112 for cauterizing tissue, making incisions, sealingblood vessels, or the like. A pneumoperitoneum device 11206 supplies gasinto the body cavity of the patient 11132 via the pneumoperitoneum tube11111 to inflate the body cavity for the purpose of securing a field ofview for the endoscope 11100 and securing a working space for thesurgeon. A recorder 11207 is a device capable of recording variousinformation associated with surgery. A printer 11208 is a device capableof printing various information associated with surgery in variousformats such as a text, an image, or a graph.

Note that the light source device 11203 supplying the irradiation lightfor photographing a surgical site to the endoscope 11100 can include awhite light source including an LED, a laser light source, or acombination thereof, for example. The output intensity and output timingof each color (each wavelength) can be controlled with high accuracy ina case where the white light source includes a combination of RGB laserlight sources, so that the white balance of a captured image can beadjusted on the light source device 11203. Moreover, in this case, animage corresponding to each of RGB can be captured by time division byprojecting the laser light from each of the RGB laser light sources ontothe observation target by time division and controlling the drive of theimaging element of the camera head 11102 in synchronization with theprojection timing. According to this method, a color image can beobtained without a color filter provided in the imaging element.

Moreover, the light source device 11203 may be controlled to change theintensity of light to be output at predetermined time intervals. Animage with a high dynamic range without so-called blackout and whiteoutcan be generated by controlling the drive of the imaging element of thecamera head 11102 in synchronization with the timing for changing theintensity of the light to acquire images by time division and combinethe images.

Furthermore, the light source device 11203 may be configured to be ableto supply light in a predetermined wavelength band corresponding tospecial light observation. The special light observation performsso-called narrow band imaging that photographs a predetermined tissuesuch as a blood vessel in a mucosal surface layer with high contrast byusing, for example, the wavelength dependence of light absorption in thebody tissue and projecting light in a narrow band narrower than thatcorresponding to the irradiation light (that is, white light) at thetime of regular observation. Alternatively, the special lightobservation may perform fluorescence observation that obtains an imageby fluorescence generated by projecting excitation light. Thefluorescence observation can observe fluorescence from a body tissue byprojecting excitation light to the body tissue (autofluorescenceobservation), or obtain a fluorescent image by performing localinjection of a reagent such as indocyanine green (ICG) into a bodytissue and at the same time projecting excitation light corresponding tothe fluorescence wavelength of the reagent to the body tissue, forexample. The light source device 11203 can be configured to be able tosupply narrowband light and/or excitation light corresponding to suchspecial light observation.

FIG. 31 is a block diagram illustrating an example of the functionalconfiguration of the camera head 11102 and the CCU 11201 illustrated inFIG. 30.

The camera head 11102 includes a lens unit 11401, an imaging unit 11402,a drive unit 11403, a communication unit 11404, and a camera headcontrol unit 11405. The CCU 11201 includes a communication unit 11411,an image processing unit 11412, and a control unit 11413. The camerahead 11102 and the CCU 11201 are connected by a transmission cable 11400to be able to communicate with each other.

The lens unit 11401 is an optical system provided at a junction with thelens barrel 11101. The observation light taken in from the tip of thelens barrel 11101 is guided to the camera head 11102 and enters the lensunit 11401. The lens unit 11401 includes a combination of a plurality oflenses including a zoom lens and a focus lens.

The imaging unit 11402 includes an imaging element. The imaging elementincluded in the imaging unit 11402 may be one (a so-called single platetype) or two or more (a so-called multi-plate type) in number. In a casewhere the imaging unit 11402 is of the multi-plate type, for example,image signals corresponding to R, G, and B may be generated by thecorresponding imaging elements and synthesized to obtain a color image.Alternatively, the imaging unit 11402 may include a pair of imagingelements for acquiring right-eye and left-eye image signals adapted forthree-dimensional (3D) display. The 3D display allows the surgeon 11131to more accurately grasp the depth of a body tissue at the surgicalsite. Note that in the case where the imaging unit 11402 is of themulti-plate type, a plurality of lens units 11401 corresponding to theimaging elements can be provided.

Moreover, the imaging unit 11402 is not necessarily provided in thecamera head 11102. For example, the imaging unit 11402 may be providedinside the lens barrel 11101 just behind the objective lens.

The drive unit 11403 includes an actuator and moves the zoom lens andthe focus lens of the lens unit 11401 by a predetermined distance alongthe optical axis under the control of the camera head control unit11405. As a result, the magnification and the focus of an image capturedby the imaging unit 11402 can be adjusted as appropriate.

The communication unit 11404 includes a communication device fortransmitting and receiving various information to and from the CCU11201. The communication unit 11404 transmits an image signal obtainedfrom the imaging unit 11402 as RAW data to the CCU 11201 via thetransmission cable 11400.

Moreover, the communication unit 11404 receives a control signal forcontrolling drive of the camera head 11102 from the CCU 11201, andsupplies the control signal to the camera head control unit 11405. Thecontrol signal includes information associated with imaging conditionssuch as information to the effect of specifying a frame rate of acaptured image, information to the effect of specifying an exposurevalue at the time of imaging, and/or information to the effect ofspecifying the magnification and focus of a captured image, for example.

Note that the above imaging conditions such as the frame rate, theexposure value, the magnification, and the focus may be specified by auser as appropriate, or may be set automatically by the control unit11413 of the CCU 11201 on the basis of the image signal acquired. In thelatter case, the so-called auto exposure (AE) function, auto focus (AF)function, and auto white balance (AWB) function are installed in theendoscope 11100.

The camera head control unit 11405 controls drive of the camera head11102 on the basis of the control signal from the CCU 11201 received viathe communication unit 11404.

The communication unit 11411 includes a communication device fortransmitting and receiving various information to and from the camerahead 11102. The communication unit 11411 receives an image signaltransmitted from the camera head 11102 via the transmission cable 11400.

Moreover, the communication unit 11411 transmits a control signal forcontrolling drive of the camera head 11102 to the camera head 11102. Theimage signal and the control signal can be transmitted bytelecommunications, optical communication, or the like.

The image processing unit 11412 performs various image processings onthe image signal which is the RAW data transmitted from the camera head11102.

The control unit 11413 performs various controls associated with imagingof the surgical site or the like by the endoscope 11100 and displayingof an image captured by the imaging of the surgical site or the like.For example, the control unit 11413 generates a control signal forcontrolling drive of the camera head 11102.

Moreover, the control unit 11413 causes the display device 11202 todisplay a captured image including the surgical site or the like on thebasis of the image signal subjected to the image processing by the imageprocessing unit 11412. At this time, the control unit 11413 mayrecognize various objects in the captured image by using various imagerecognition techniques. For example, the control unit 11413 detects theshape, color, or the like of the edge of an object included in thecaptured image to be able to recognize a surgical tool such as aforceps, a specific body site, bleeding, mist at the time of using theenergy treatment tool 11112, and the like. When causing the displaydevice 11202 to display the captured image, the control unit 11413 mayuse a result of the recognition and superimpose various surgery supportinformation on the image of the surgical site. The surgery supportinformation superimposed on the image and presented to the surgeon 11131allows the surgeon 11131 to have reduced burden and proceed with thesurgery reliably.

The transmission cable 11400 connecting the camera head 11102 to the CCU11201 is an electric signal cable compatible with communication ofelectrical signals, an optical fiber compatible with opticalcommunication, or a composite cable thereof.

Here, although the communication is performed by wire using thetransmission cable 11400 in the illustrated example, the communicationbetween the camera head 11102 and the CCU 11201 may be performedwirelessly.

An example of the endoscopic surgical system to which the technologyaccording to the present disclosure can be applied has been describedabove. The technology according to the present disclosure can be appliedto the endoscope 11100, the camera head 11102 (the imaging unit 11402thereof), the CCU 11201 (the image processing unit 11412 thereof), andthe like among the configurations described above, for example. Theapplication of the technology according to the present disclosure asdescribed above can obtain an image in which blurring is reliablycorrected, so that the surgeon can see the surgical site withreliability.

Note that although the endoscopic surgical system has been describedherein as an example, the technology according to the present disclosuremay also be applied to a microscopic surgery system or the like, forexample.

<Example of Application to Mobile Body>

The technology according to the present disclosure (the presenttechnology) can be applied to various products. For example, thetechnology according to the present disclosure may be implemented as adevice mounted on a mobile body of any type such as a vehicle, anelectric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, apersonal mobility, an airplane, a drone, a ship, or a robot.

FIG. 32 is a block diagram illustrating an example of the schematicconfiguration of a vehicle control system which is an example of amobile body control system to which the technology according to thepresent disclosure can be applied.

A vehicle control system 12000 includes a plurality of electroniccontrol units connected via a communication network 12001. In theexample illustrated in FIG. 32, the vehicle control system 12000includes a drive system control unit 12010, a body system control unit12020, an extra-vehicle information detection unit 12030, anintra-vehicle information detection unit 12040, and an integratedcontrol unit 12050. Moreover, as a functional configuration of theintegrated control unit 12050, a microcomputer 12051, a sound-imageoutput unit 12052, and an on-board network interface (I/F) 12053 areillustrated.

The drive system control unit 12010 controls the operation of a deviceassociated with a drive system of a vehicle according to variousprograms. For example, the drive system control unit 12010 functions asa controller of a driving force generator such as an internal combustionengine or a driving motor for generating the driving force of thevehicle, a driving force transmitting mechanism for transmitting thedriving force to the wheels, a steering mechanism for adjusting thesteering angle of the vehicle, a braking device for generating thebraking force of the vehicle, and the like.

The body system control unit 12020 controls the operation of variousdevices installed to the vehicle body according to various programs. Forexample, the body system control unit 12020 functions as a controller ofa keyless entry system, a smart key system, a power window device, orvarious lamps such as a head lamp, a back lamp, a brake lamp, a turnsignal, or a fog lamp. In this case, the body system control unit 12020can receive input of radio waves transmitted from a portable devicesubstituted for a key or signals of various switches. The body systemcontrol unit 12020 receives input of these radio waves or signals tocontrol the door lock device, power window device, lamps, or the like ofthe vehicle.

The extra-vehicle information detection unit 12030 detects informationregarding the outside of the vehicle on which the vehicle control system12000 is mounted. The extra-vehicle information detection unit 12030 isconnected to an imaging unit 12031, for example. The extra-vehicleinformation detection unit 12030 causes the imaging unit 12031 tocapture an image of the outside of the vehicle and receives the imagecaptured. The extra-vehicle information detection unit 12030 may performobject detection processing or distance detection processing for aperson, a vehicle, an obstacle, a sign, a character on a road surface,or the like on the basis of the image received.

The imaging unit 12031 is an optical sensor that receives light andoutputs an electrical signal corresponding to the amount of lightreceived. The imaging unit 12031 can output the electrical signal as animage or as ranging information. Moreover, the light received by theimaging unit 12031 may be visible light or invisible light such asinfrared light.

The intra-vehicle information detection unit 12040 detects informationregarding the inside of the vehicle. The intra-vehicle informationdetection unit 12040 is connected to a driver condition detection unit12041 for detecting the condition of a driver, for example. The drivercondition detection unit 12041 includes a camera that images the driver,for example, and the intra-vehicle information detection unit 12040 maycalculate a degree of fatigue or degree of concentration of the driveror may determine whether the driver is dozing off on the basis of thedetection information input from the driver condition detection unit12041.

The microcomputer 12051 calculates a control target value of the drivingforce generator, the steering mechanism, or the braking device on thebasis of the information regarding the inside or outside of the vehicleacquired by the extra-vehicle information detection unit 12030 or theintra-vehicle information detection unit 12040, thereby being able tooutput a control command to the drive system control unit 12010. Forexample, the microcomputer 12051 can perform cooperative control for thepurpose of implementing the function of an advanced driver assistancesystem (ADAS) including collision avoidance or impact mitigation for thevehicle, travel following a vehicle ahead, constant speed travel, or avehicle collision warning based on the distance between vehicles, awarning for the vehicle going off the lane, or the like.

Moreover, the microcomputer 12051 controls the driving force generator,the steering mechanism, the braking device, or the like on the basis ofinformation regarding the surroundings of the vehicle acquired by theextra-vehicle information detection unit 12030 or the intra-vehicleinformation detection unit 12040, thereby being able to performcooperative control for the purpose of automated driving or the likewith which a vehicle travels autonomously without depending on thedriver's operation.

Furthermore, the microcomputer 12051 can output a control command to thebody system control unit 12020 on the basis of the information regardingthe outside of the vehicle acquired by the extra-vehicle informationdetection unit 12030. For example, the microcomputer 12051 can performcooperative control for the purpose of anti-glare such as switching fromhigh beam to low beam by controlling the head lamp depending on theposition of a vehicle ahead or an oncoming vehicle detected by theextra-vehicle information detection unit 12030.

The sound-image output unit 12052 transmits an output signal of at leastone of sound or image to an output device that can visually or aurallyprovide notification of information to a passenger of the vehicle or theoutside of the vehicle. The example of FIG. 32 illustrates an audiospeaker 12061, a display unit 12062, and an instrument panel 12063 asthe output devices. The display unit 12062 may include at least one ofan on-board display or a head-up display, for example.

FIG. 33 is a diagram illustrating an example of the installationposition of the imaging unit 12031.

In FIG. 33, a vehicle 12100 includes imaging units 12101, 12102, 12103,12104, and 12105 as the imaging unit 12031.

The imaging units 12101, 12102, 12103, 12104, and 12105 are installed atpositions such as a front nose, side mirrors, a rear bumper, a backdoor, and an upper part of the windshield in the passenger compartmentof the vehicle 12100, for example. The imaging unit 12101 installed atthe front nose and the imaging unit 12105 installed in the upper part ofthe windshield in the passenger compartment mainly acquire an image ofthe area ahead of the vehicle 12100. The imaging units 12102 and 12103installed on the side mirrors mainly acquire images of the sides of thevehicle 12100. The imaging unit 12104 installed on the rear bumper orthe back door mainly acquires an image of the area behind the vehicle12100. The image of the area ahead of the vehicle acquired by theimaging units 12101 and 12105 is mainly used for detecting a vehicleahead or a pedestrian, an obstacle, a traffic light, a traffic sign, alane, or the like.

Note that FIG. 33 illustrates an example of the imaging range of theimaging units 12101 to 12104. An imaging range 12111 indicates theimaging range of the imaging unit 12101 installed at the front nose,imaging ranges 12112 and 12113 indicate the imaging ranges of theimaging units 12102 and 12103 installed on the side mirrors,respectively, and an imaging range 12114 indicates the imaging range ofthe imaging unit 12104 installed on the rear bumper or the back door.For example, a bird's eye view image of the vehicle 12100 viewed fromabove is obtained by superimposing image data captured by the imagingunits 12101 to 12104.

At least one of the imaging units 12101 to 12104 may have a function ofacquiring distance information. For example, at least one of the imagingunits 12101 to 12104 may be a stereo camera including a plurality ofimaging elements, or may be an imaging element having pixels for phasedifference detection.

For example, on the basis of the distance information obtained from theimaging units 12101 to 12104, the microcomputer 12051 finds the distanceto each three-dimensional object within the imaging ranges 12111 to12114 and a temporal change in the distance (relative speed with respectto the vehicle 12100), thereby being able to particularly extract, as avehicle ahead, a three-dimensional object closest on the path of travelof the vehicle 12100 and traveling at a predetermined speed (forexample, 0 km/h or faster) in substantially the same direction as thatof the vehicle 12100. Moreover, the microcomputer 12051 can set inadvance the distance between vehicles to be secured behind a vehicleahead, thereby being able to perform automatic brake control (includingfollow-up stop control), automatic acceleration control (includingfollow-up start control), and the like. The microcomputer can thusperform the cooperative control for the purpose of automated driving orthe like with which a vehicle travels autonomously without depending onthe driver's operation.

For example, on the basis of the distance information obtained from theimaging units 12101 to 12104, the microcomputer 12051 classifiesthree-dimensional object data associated with a three-dimensional objectinto a two-wheeled vehicle, a standard sized vehicle, a large sizedvehicle, a pedestrian, and other three-dimensional objects such as autility pole, and extracts the data for use in automatic avoidance of anobstacle. For example, the microcomputer 12051 identifies an obstacle inthe vicinity of the vehicle 12100 as an obstacle that can be visuallyrecognized by the driver of the vehicle 12100 or an obstacle that cannoteasily be visually recognized by the driver. Then, the microcomputer12051 determines the risk of collision indicating the degree of risk ofcollision with each obstacle, and under circumstances where there is apossibility of collision with the risk of collision higher than or equalto a set value, the microcomputer can perform driver assistance to avoidcollision by outputting an alarm to the driver via the audio speaker12061 and/or the display unit 12062 or performing forced deceleration orevasive steering via the drive system control unit 12010.

At least one of the imaging units 12101 to 12104 may be an infraredcamera for detecting infrared light. For example, the microcomputer12051 can recognize a pedestrian by determining whether or not apedestrian is present in images captured by the imaging units 12101 to12104. Such pedestrian recognition is performed by a procedure ofextracting feature points in the images captured by the imaging units12101 to 12104 as infrared cameras, for example, and a procedure ofperforming pattern matching on a series of feature points indicating theoutline of an object and determining whether or not the objectcorresponds to a pedestrian. If the microcomputer 12051 determines thata pedestrian is present in the images captured by the imaging units12101 to 12104 and recognizes the pedestrian, the sound-image outputunit 12052 controls the display unit 12062 such that a rectangularcontour for emphasis is superimposed and displayed on the pedestrianbeing recognized. The sound-image output unit 12052 may also control thedisplay unit 12062 to display an icon or the like indicating thepedestrian at a desired position.

An example of the vehicle control system to which the technologyaccording to the present disclosure can be applied has been describedabove. The technology according to the present disclosure can be appliedto the imaging unit 12031 and the like among the configurationsdescribed above, for example. The application of the technologyaccording to the present disclosure as described above can obtain animage in which blurring is reliably corrected.

Note that the present technology can also be embodied in the followingconfigurations.

(1)

An imaging device including:

an imaging unit that captures an image of a subject via an opticalsystem collecting light from the subject;

a drive control unit that controls drive of at least one of the opticalsystem or the imaging unit by finding, on the basis of motion of theimaging unit that is physically detected, an amount of movement at thetime at least one of the optical system or the imaging unit is movedrelative to another to optically correct blurring appearing in the imagecaptured by the imaging unit; and

a signal processing unit that performs signal processing for correctingan effect of the motion of the imaging unit on the image according to aposition conversion function based on position information and motioninformation synchronized with each coordinate on the image, on the basisof the position information obtained by detecting a position of theoptical system or the imaging unit driven according to the control bythe drive control unit and the motion information indicating the motionof the imaging unit physically detected.

(2)

The imaging device according to (1) above, further including

a logic unit that supplies the position information, the motioninformation, and timing information to the signal processing unittogether with the image captured by the imaging unit, the timinginformation indicating a timing for synchronizing the positioninformation and the motion information with a coordinate on the image.

(3)

The imaging device according to (2) above, in which

the logic unit adds the position information and the motion informationtogether with the timing information to the image and outputs the imageto which the information is added.

(4)

The imaging device according to (2) above, in which

the logic unit outputs, as the timing information, a piece ofinformation indicating a position in a vertical direction on the imagein association with the position information and the motion informationby the line of the position in the vertical direction.

(5)

The imaging device according to any of (2) to (4) above, furtherincluding

an image sensor in which the imaging unit and the logic unit arestacked, in which

the image sensor supplies the position information, the motioninformation, and the timing information together with the image to thesignal processing unit.

(6)

The imaging device according to any of (1) to (5) above, furtherincluding

a drive unit that drives at least one of the optical system or theimaging unit according to the amount of movement found by the drivecontrol unit, detects the position of the optical system or the imagingunit being driven, and supplies the position information to the drivecontrol unit.

(7)

The imaging device according to any of (2) to (6) above, furtherincluding

a detection unit that physically detects the motion of the imaging unitand supplies the motion information to the drive control unit, in which

the drive control unit supplies the position information and the motioninformation to the logic unit.

(8)

The imaging device according to any of (2) to (5) above, in which

the logic unit generates and supplies, to the drive control unit,control information instructing execution or termination of the opticalcorrection according to an exposure timing at which the imaging unitperforms exposure, and

on the basis of the control information, the drive control unit controlsdrive of at least one of the optical system or the imaging unit to causethe optical system or the imaging unit to perform optical correction onblurring that appears in the image captured by the imaging unit during aperiod of execution of the optical correction, and controls drive tobring the optical system or the imaging unit back to a center positionduring termination of the optical correction.

(9)

The imaging device according to (8) above, in which

in a case where a period for which the control information instructstermination of the optical correction is shorter than time required tobring the optical system or the imaging unit back to the centerposition, the drive control unit controls drive to move the opticalsystem or the imaging unit toward the center to the extent that theoptical system or the imaging unit can be moved within the period.

(10)

The imaging device according to any of (1) to (9) above, in which

the signal processing unit performs the signal processing at the time ofperforming processing that outputs a still image obtained bysuperimposing a plurality of sequential images captured sequentiallywith a short exposure time.

(11)

A solid-state image sensor including:

an imaging unit that captures an image of a subject via an opticalsystem collecting light from the subject; and a logic unit that performsprocessing of adding, to the image captured by the imaging unit,position information obtained by detecting a position of the opticalsystem or the imaging unit driven according to control by a drivecontrol unit and motion information indicating motion of the imagingunit physically detected, and outputs the image to which the positioninformation and the motion information is added to a signal processingunit that performs signal processing for correcting an effect of themotion of the imaging unit on the image according to a positionconversion function based on the position information and the motioninformation synchronized with each coordinate on the image on the basisof the position information and the motion information, the drivecontrol unit controlling drive of at least one of the optical system orthe imaging unit by finding, on the basis of the motion of the imagingunit that is physically detected, an amount of movement at the time atleast one of the optical system or the imaging unit is moved relative toanother to optically correct blurring appearing in the image captured bythe imaging unit.

(12)

A camera module including:

an optical system that collects light from a subject;

an imaging unit that captures an image of the subject via the opticalsystem;

a drive control unit that controls drive of at least one of the opticalsystem or the imaging unit by finding, on the basis of motion of theimaging unit that is physically detected, an amount of movement at thetime at least one of the optical system or the imaging unit is movedrelative to another to optically correct blurring appearing in the imagecaptured by the imaging unit; and

a logic unit that supplies position information, motion information, andtiming information indicating a timing for synchronizing the positioninformation and the motion information with a coordinate on the image toa signal processing unit together with the image captured by the imagingunit, the signal processing unit performing signal processing forcorrecting an effect of the motion of the imaging unit on the imageaccording to a position conversion function based on the positioninformation and the motion information synchronized with each coordinateon the image on the basis of the position information obtained bydetecting a position of the optical system or the imaging unit drivenaccording to the control by the drive control unit and the motioninformation indicating the motion of the imaging unit physicallydetected.

(13)

A drive control unit that controls drive of at least one of an opticalsystem or an imaging unit by finding, on the basis of physicallydetected motion of the imaging unit capturing an image of a subject viathe optical system collecting light from the subject, an amount ofmovement at the time at least one of the optical system or the imagingunit is moved relative to another to optically correct blurringappearing in the image captured by the imaging unit, and

supplies position information, which is obtained by detecting a positionof the optical system or the imaging unit driven according to thecontrol, and motion information, which indicates the physically detectedmotion of the imaging unit, to a logic unit that performs processing ofadding the position information and the motion information to the imagecaptured by the imaging unit and outputs the image to which the positioninformation and the motion information is added to a signal processingunit that performs signal processing for correcting an effect of themotion of the imaging unit on the image according to a positionconversion function based on the position information and the motioninformation synchronized with each coordinate on the image on the basisof the position information and the motion information.

(14)

An imaging method including the steps of:

controlling drive of at least one of an optical system or an imagingunit by finding, on the basis of physically detected motion of theimaging unit capturing an image of a subject via the optical systemcollecting light from the subject, an amount of movement at the time atleast one of the optical system or the imaging unit is moved relative toanother to optically correct blurring appearing in the image captured bythe imaging unit; and

performing signal processing for correcting an effect of the motion ofthe imaging unit on the image according to a position conversionfunction based on position information and motion informationsynchronized with each coordinate on the image on the basis of theposition information obtained by detecting a position of the opticalsystem or the imaging unit driven according to the control and themotion information indicating the physically detected motion of theimaging unit.

(15)

The imaging method according to (14) above, further including

a step of performing processing that adds the position information andthe motion information to the image captured by the imaging unittogether with timing information indicating a position in a verticaldirection on the image subjected to exposure at a timing of acquisitionof the position information and the motion information.

Note that the present embodiment is not limited to the aforementionedembodiment, where various modifications can be made without departingfrom the scope of the present disclosure.

REFERENCE SIGNS LIST

-   11 Imaging device-   12 Lens unit-   13 Image sensor-   14 Motion sensor-   15 OIS driver-   16 OIS actuator-   17 Signal processing circuit-   18 Display-   19 Recording medium-   21 Imaging unit-   22 Logic unit

1. An imaging device comprising: an imaging unit that captures an imageof a subject via an optical system collecting light from the subject; adrive control unit that controls drive of at least one of the opticalsystem or the imaging unit by finding, on a basis of motion of theimaging unit that is physically detected, an amount of movement at thetime at least one of the optical system or the imaging unit is movedrelative to another to optically correct blurring appearing in the imagecaptured by the imaging unit; and a signal processing unit that performssignal processing for correcting an effect of the motion of the imagingunit on the image according to a position conversion function based onposition information and motion information synchronized with eachcoordinate on the image, on a basis of the position information obtainedby detecting a position of the optical system or the imaging unit drivenaccording to the control by the drive control unit and the motioninformation indicating the motion of the imaging unit physicallydetected.
 2. The imaging device according to claim 1, further comprisinga logic unit that supplies the position information, the motioninformation, and timing information to the signal processing unittogether with the image captured by the imaging unit, the timinginformation indicating a timing for synchronizing the positioninformation and the motion information with a coordinate on the image.3. The imaging device according to claim 2, wherein the logic unit addsthe position information and the motion information together with thetiming information to the image and outputs the image to which theinformation is added.
 4. The imaging device according to claim 2,wherein the logic unit outputs, as the timing information, a piece ofinformation indicating a position in a vertical direction on the imagein association with the position information and the motion informationby the line of the position in the vertical direction.
 5. The imagingdevice according to claim 2, further comprising an image sensor in whichthe imaging unit and the logic unit are stacked, wherein the imagesensor supplies the position information, the motion information, andthe timing information together with the image to the signal processingunit.
 6. The imaging device according to claim 1, further comprising adrive unit that drives at least one of the optical system or the imagingunit according to the amount of movement found by the drive controlunit, detects the position of the optical system or the imaging unitbeing driven, and supplies the position information to the drive controlunit.
 7. The imaging device according to claim 2, further comprising adetection unit that physically detects the motion of the imaging unitand supplies the motion information to the drive control unit, whereinthe drive control unit supplies the position information and the motioninformation to the logic unit.
 8. The imaging device according to claim2, wherein the logic unit generates and supplies, to the drive controlunit, control information instructing execution or termination of theoptical correction according to an exposure timing at which the imagingunit performs exposure, and on a basis of the control information, thedrive control unit controls drive of at least one of the optical systemor the imaging unit to cause the optical system or the imaging unit toperform optical correction on blurring that appears in the imagecaptured by the imaging unit during a period of execution of the opticalcorrection, and controls drive to bring the optical system or theimaging unit back to a center position during termination of the opticalcorrection.
 9. The imaging device according to claim 7, wherein in acase where a period for which the control information instructstermination of the optical correction is shorter than time required tobring the optical system or the imaging unit back to a center position,the drive control unit controls drive to move the optical system or theimaging unit toward the center to the extent that the optical system orthe imaging unit can be moved within the period.
 10. The imaging deviceaccording to claim 1, wherein the signal processing unit performs thesignal processing at the time of performing processing that outputs astill image obtained by superimposing a plurality of sequential imagescaptured sequentially with a short exposure time.
 11. A solid-stateimage sensor comprising: an imaging unit that captures an image of asubject via an optical system collecting light from the subject; and alogic unit that performs processing of adding, to the image captured bythe imaging unit, position information obtained by detecting a positionof the optical system or the imaging unit driven according to control bya drive control unit and motion information indicating motion of theimaging unit physically detected, and outputs the image to which theposition information and the motion information is added to a signalprocessing unit that performs signal processing for correcting an effectof the motion of the imaging unit on the image according to a positionconversion function based on the position information and the motioninformation synchronized with each coordinate on the image on a basis ofthe position information and the motion information, the drive controlunit controlling drive of at least one of the optical system or theimaging unit by finding, on a basis of the motion of the imaging unitthat is physically detected, an amount of movement at the time at leastone of the optical system or the imaging unit is moved relative toanother to optically correct blurring appearing in the image captured bythe imaging unit.
 12. A camera module comprising: an optical system thatcollects light from a subject; an imaging unit that captures an image ofthe subject via the optical system; a drive control unit that controlsdrive of at least one of the optical system or the imaging unit byfinding, on a basis of motion of the imaging unit that is physicallydetected, an amount of movement at the time at least one of the opticalsystem or the imaging unit is moved relative to another to opticallycorrect blurring appearing in the image captured by the imaging unit;and a logic unit that supplies position information, motion information,and timing information indicating a timing for synchronizing theposition information and the motion information with a coordinate on theimage to a signal processing unit together with the image captured bythe imaging unit, the signal processing unit performing signalprocessing for correcting an effect of the motion of the imaging unit onthe image according to a position conversion function based on theposition information and the motion information synchronized with eachcoordinate on the image on a basis of the position information obtainedby detecting a position of the optical system or the imaging unit drivenaccording to the control by the drive control unit and the motioninformation indicating the motion of the imaging unit physicallydetected.
 13. A drive control unit that controls drive of at least oneof an optical system or an imaging unit by finding, on a basis ofphysically detected motion of the imaging unit capturing an image of asubject via the optical system collecting light from the subject, anamount of movement at the time at least one of the optical system or theimaging unit is moved relative to another to optically correct blurringappearing in the image captured by the imaging unit, and suppliesposition information, which is obtained by detecting a position of theoptical system or the imaging unit driven according to the control, andmotion information, which indicates the physically detected motion ofthe imaging unit, to a logic unit that performs processing of adding theposition information and the motion information to the image captured bythe imaging unit and outputs the image to which the position informationand the motion information is added to a signal processing unit thatperforms signal processing for correcting an effect of the motion of theimaging unit on the image according to a position conversion functionbased on the position information and the motion informationsynchronized with each coordinate on the image on a basis of theposition information and the motion information.
 14. An imaging methodcomprising the steps of: controlling drive of at least one of an opticalsystem or an imaging unit by finding, on a basis of physically detectedmotion of the imaging unit capturing an image of a subject via theoptical system collecting light from the subject, an amount of movementat the time at least one of the optical system or the imaging unit ismoved relative to another to optically correct blurring appearing in theimage captured by the imaging unit; and performing signal processing forcorrecting an effect of the motion of the imaging unit on the imageaccording to a position conversion function based on positioninformation and motion information synchronized with each coordinate onthe image on a basis of the position information obtained by detecting aposition of the optical system or the imaging unit driven according tothe control and the motion information indicating the physicallydetected motion of the imaging unit.
 15. The imaging method according toclaim 14, further comprising a step of performing processing that addsthe position information and the motion information to the imagecaptured by the imaging unit together with timing information indicatinga position in a vertical direction on the image subjected to exposure ata timing of acquisition of the position information and the motioninformation.